Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added job scheduling page #23

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Added job scheduling page #23

wants to merge 7 commits into from

Conversation

Med16-11
Copy link
Collaborator

@Med16-11 Med16-11 commented Jun 25, 2023

screen-capture.mp4

This PR is dependent on the following PRs:

ceph/teuthology-api#51
ceph/teuthology#1924

Testing this PR

please checkout -b the PRs above for teuthology and teuthology-api.

And in teuthology-api do

diff --git a/setup.cfg b/setup.cfg
index babeb07..26154f1 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -56,7 +56,7 @@ install_requires =
     itsdangerous
     pydantic-settings
     python-dotenv
-    teuthology @ git+https://github.com/ceph/teuthology#egg=teuthology[test]
+    teuthology @ git+https://github.com/ceph/teuthology@wip-ksirivad-teuth-suite-exception

@kamoltat kamoltat self-requested a review June 26, 2023 14:37
@kamoltat kamoltat added the feature New feature or request label Jun 27, 2023
@kamoltat
Copy link
Member

@Med16-11 always rebase against latest main branch before filing a PR, or else there will likely be merge conflicts like what you see right now.

</button>
</div>
</div>
{/* <div>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check for things like commented code before committing :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I actually thought that we can use the commented code later to ensure that buttons are working properly and can write the logic in functions later.

@kamoltat
Copy link
Member

Okay so a few things we can improve:

  1. The key and value column should be a dynamic table where when you input data into the last row, it will automatically generate a new row.
  2. key and value should be an input form ( currently it is static)
  3. When hover on to a row, you should be able to delete it, similar to POSTMAN where a little trash icon will appear and you can just delete the row.
  4. The top box which displays the teutholgoy-suite command should automatically generate the command based on the kv that is given in table.
  5. run and dry-run button should have red and blue colors.

@kamoltat kamoltat linked an issue Jun 29, 2023 that may be closed by this pull request
4 tasks
@Med16-11
Copy link
Collaborator Author

Med16-11 commented Jun 30, 2023

Okay so a few things we can improve:

  1. The key and value column should be a dynamic table where when you input data into the last row, it will automatically generate a new row.
  2. key and value should be an input form ( currently it is static)
  3. When hover on to a row, you should be able to delete it, similar to POSTMAN where a little trash icon will appear and you can just delete the row.
  4. The top box which displays the teutholgoy-suite command should automatically generate the command based on the kv that is given in table.
  5. run and dry-run button should have red and blue colors.

Got it.

@Med16-11 Med16-11 closed this Jun 30, 2023
@Med16-11
Copy link
Collaborator Author

It seems like I have closed the PR accidentally. Could you please open it @kamoltat

@kamoltat kamoltat reopened this Jun 30, 2023
@kamoltat
Copy link
Member

Thanks you for filling the change,

here are some of the things I think can be improved:

  1. The button, you should use import Button from "@mui/material/Button"; the color for run button can be used as color="error" for now. For dry-run button try to find a blue color that matches the color="error".
  2. The command box only gets updated once you press the run button, what we should do is to have the command box update every-time we check/uncheck the box for each key. This also applies when we delete a row that is checked.
  3. Instead of generating a new column for the delete row button, we can have a small trash/bin icon inside the value column for each row, similar to how POSTMAN did it.

@Med16-11
Copy link
Collaborator Author

image
It looks like this now.

@kamoltat kamoltat self-requested a review July 18, 2023 13:12
Copy link
Member

@kamoltat kamoltat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Med16-11
Thanks for the change, I think with the table color being black it is hard to see in dark mode, might be worth also changing the border to white or light grey when in dark mode.
image

@kamoltat
Copy link
Member

kamoltat commented Jul 18, 2023

I realized that you have a merge commit, we can avoid that by doing:

git fetch upstream

git rebase upstream/main

Also, to keep the commits clean in your PR, you should try to squash your commits in to 1 commit whenever the change between the two commits serve the same purpose.

You can do this by:

commit your new change:
git add <file>
write some random commit message
git commit -s
Now lets say you have N new commits on top of main you do:
git rebase -HEAD~N

In the interactive shell put an s

pick 02efde5 resolved conflits
pick 1b725ad resolved other conflits
pick 9374fad added the suggested changes
pick dfd9165 pages/Job/index.js: fix job started time
pick 15386e1 Implement kill-run button using mock API endpoint
s 7a3edd9 new changes added

Then it will redirect you to commit message , just edit the commit by deleting the random commit message you wrote and keep/edit the original commit message that you a squashing to.

@kamoltat
Copy link
Member

Pending merge conflict to be resolved

@Med16-11
Copy link
Collaborator Author

Pending merge conflict to be resolved

On it.

@Med16-11
Copy link
Collaborator Author

image
That's how table looks now in dark mode.

@Med16-11
Copy link
Collaborator Author

Med16-11 commented Jul 26, 2023

I'll be squashing all the commits after receiving your reviews.

@kamoltat
Copy link
Member

@Med16-11 Thanks for the table change the border looks good now for dark mode.

@kamoltat
Copy link
Member

@Med16-11 I've noticed that:

  1. The input box for the key column cannot be edited?
  2. If we tick the checkbox for a bunch keys/values and delete it, the command box up top still retains the value of the deleted row.

@Med16-11
Copy link
Collaborator Author

@Med16-11 I've noticed that:

  1. The input box for the key column cannot be edited?
  2. If we tick the checkbox for a bunch keys/values and delete it, the command box up top still retains the value of the deleted row.

Okay, I'll look into it.

@kamoltat
Copy link
Member

kamoltat commented Aug 2, 2023

@Med16-11 thanks for your new change,
you're very close,

  1. so I think the key column is still unable to be inputed.
  2. Trash Icon in dark mode cannot be seen since it is white and when the cursor is on the row, it becomes white as well

@VallariAg
Copy link
Member

@kamoltat @zmc
The schema for /suite endpoint on teuthology-api is defined and restrictive. Since the API does not allow the freedom to add any other args to the teuthology-suite command, the KEY column's can be plain text (listing all possible args) instead of input fields. This would let the user know about the restrictions of key names. Otherwise we would need to check the key names before requesting teuthology-api and throw an error if its not supported.
What do you think?

@Med16-11
Good work! I know it's WIP, just letting you some of the issues:

  1. If I edit a row after clicking the "check", the command on top does not edit.
  2. Command input at top and the form below are out of sync if something is directly added to the command input. They need to be synced as it would cause confusion for the user about which args would be considered for the run.

@zmc
Copy link
Member

zmc commented Aug 7, 2023

@VallariAg If I understand you correctly, you're suggesting we not allow/require the user to input the flag names in the "Key" column, and instead list all the available options and allow the user to fill in the ones they need. I think I agree with that. We could populate a Select and allow adding items to the table from that list.

This reminds me: at some point we'll want to be populating default values; perhaps we should add an endpoint to teuthology-api that provides these.

@VallariAg
Copy link
Member

@zmc that sounds perfect. I'll open an issue for adding an endpoint for default values on teuthology-api and using it on pulpito-ng.

In teuthology meeting, we also discussed to disable editing on the command input above the table. We can focus on scheduling runs with the form table for now. I'll open a new issue for later enabling that feature.

@kamoltat
Copy link
Member

kamoltat commented Aug 9, 2023

@zmc @VallariAg
Okay so I think I'm fine with having all the suite keys as an out of the box experience. But from a Teuthology user perspective I only use around 10 - 13 out of 50+ arguments. If I were to use pulpito-ng to schedule jobs I would probably like to have only 10-13 keys in my key column with like at least 2-3 keys being selected by default. Therefore, I think we should keep the key delete functionality and key add button.

@kamoltat
Copy link
Member

kamoltat commented Feb 1, 2024

NOTE: I always forget to drop the eb58790
since the PR that has this commit hasn't been merged yet. So reviewer please help watch out for this hot reload commit.

Nevermind, the PR for that commit is merged so it is no longer cherry-picked to my dev local branch

@kamoltat kamoltat force-pushed the medhavi branch 2 times, most recently from 5cefc09 to aff5afd Compare February 14, 2024 15:06
@kamoltat
Copy link
Member

kamoltat commented Feb 14, 2024

In order, to test this PR properly, we need to modify the dry_run args in teuthology_api

diff --git a/src/teuthology_api/services/suite.py b/src/teuthology_api/services/suite.py
index 8f1d0f7..af29aaa 100644
--- a/src/teuthology_api/services/suite.py
+++ b/src/teuthology_api/services/suite.py
@@ -22,8 +22,8 @@ def run(args, dry_run: bool, send_logs: bool, access_token: str):
         )
     try:
         args["--timestamp"] = datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
-        if dry_run:
-            args["--dry-run"] = True
+        if args["--dry-run"] == True:
+            log.info("This run is a dry run")
             logs = logs_run(teuthology.suite.main, args)
             return {"run": {}, "logs": logs}

(Edit: ceph/teuthology-api#48 resolved this)

Med16-11 and others added 5 commits March 7, 2024 14:24
Signed-off-by: Med16-11 <[email protected]>

resolved conflits
1. Rewrote the components with Material UI
2. Improved OnListener logics
3. Improved how we store data in useLocalStorage.

Signed-off-by: Kamoltat Sirivadhna <[email protected]>
…ling suites

Use axios post request to for run, dry-run and force-priority

Signed-off-by: Kamoltat Sirivadhna <[email protected]>
@kamoltat
Copy link
Member

screen-capture.mp4

@kamoltat
Copy link
Member

kamoltat commented May 10, 2024

ran into this error:

Run npm run lint

> [email protected] lint
> eslint src

/home/runner/work/pulpito-ng/pulpito-ng/src/lib/teuthologyAPI.ts
  30:22  error  React Hook "useUserData" is called in function "doSchedule" that is neither a React function component nor a custom React Hook function. React component names must start with an uppercase letter. React Hook names must start with the word "use"  react-hooks/rules-of-hooks

Error: Process completed with exit code 1.

Therefore, I decided to change the name doSchedule to useSchedule

schedule feature uses useMutation to deal with cases
like onSuccess, onError and isLoading.

CircularProgress is added when button is clicked
and mutation is in the state of isLoading.

Signed-off-by: Kamoltat Sirivadhna <[email protected]>
1. After a run is scheduled, user
will be able to see the results including
the logs from teuthology-suite. Only success runs will
show all logs from teuthology-suite, failed runs
will only return the exception with descriptive failure
reasons.

2. Added Warning Alerts for 0 jobs scheduled in a run

3. Added more arguments that can be use to schedule run

Signed-off-by: Kamoltat Sirivadhna <[email protected]>
@VallariAg
Copy link
Member

Hey @kamoltat,
I was testing this PR and I keep running into the following issue when I send a dry-run schedule request:
image

docker-compose-teuthology_api-1  | INFO:     192.168.96.1:47482 - "POST /suite?logs=true HTTP/1.1" 307 Temporary Redirect
docker-compose-teuthology_api-1  | INFO:     192.168.96.1:47468 - "OPTIONS /suite/?logs=true HTTP/1.1" 200 OK
docker-compose-teuthology_api-1  | 2024-06-10 11:08:12,547.547 INFO:root:teuthology version: 1.1.1.dev689+g3f548c60
docker-compose-teuthology_api-1  | 2024-06-10 11:08:12,548.548 INFO:teuthology.suite:Using random seed=3261
docker-compose-teuthology_api-1  | 2024-06-10 11:08:12,548.548 INFO:teuthology.suite.run:kernel sha1: distro
docker-compose-teuthology_api-1  | Process Process-1:1:
docker-compose-teuthology_api-1  | Traceback (most recent call last):
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/suite/run.py", line 200, in choose_ceph_hash
docker-compose-teuthology_api-1  |     ceph_hash = util.git_ls_remote(
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/suite/util.py", line 163, in git_ls_remote
docker-compose-teuthology_api-1  |     return repo_utils.ls_remote(url, branch)
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/repo_utils.py", line 76, in ls_remote
docker-compose-teuthology_api-1  |     raise GitError(e.output) from None
docker-compose-teuthology_api-1  | teuthology.exceptions.GitError: b"fatal: could not read Username for 'https://github.com': No such device or address\n"
docker-compose-teuthology_api-1  | 
docker-compose-teuthology_api-1  | During handling of the above exception, another exception occurred:
docker-compose-teuthology_api-1  | 
docker-compose-teuthology_api-1  | Traceback (most recent call last):
docker-compose-teuthology_api-1  |   File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
docker-compose-teuthology_api-1  |     self.run()
docker-compose-teuthology_api-1  |   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
docker-compose-teuthology_api-1  |     self._target(*self._args, **self._kwargs)
docker-compose-teuthology_api-1  |   File "/teuthology_api/src/teuthology_api/services/helpers.py", line 62, in _execute_with_logs
docker-compose-teuthology_api-1  |     job_count = func(args)
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/suite/__init__.py", line 153, in main
docker-compose-teuthology_api-1  |     run = Run(conf)
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/suite/run.py", line 57, in __init__
docker-compose-teuthology_api-1  |     self.base_config = self.create_initial_config()
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/suite/run.py", line 92, in create_initial_config
docker-compose-teuthology_api-1  |     ceph_hash = self.choose_ceph_hash()
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/suite/run.py", line 203, in choose_ceph_hash
docker-compose-teuthology_api-1  |     raise util.schedule_fail(message=str(e), name=self.name, dry_run=self.args.dry_run) from None
docker-compose-teuthology_api-1  |   File "/usr/local/lib/python3.8/dist-packages/teuthology/suite/util.py", line 77, in schedule_fail
docker-compose-teuthology_api-1  |     raise ScheduleFailError(message, name)
docker-compose-teuthology_api-1  | teuthology.exceptions.ScheduleFailError: Scheduling VallariAg-2024-06-10_11:08:11-teuthology:no-ceph-main-distro-default-smithi failed: b"fatal: could not read Username for 'https://github.com': No such device or address\n"
docker-compose-teuthology_api-1  | 2024-06-10 11:08:14,164.164 ERROR:teuthology_api.services.helpers:Scheduling VallariAg-2024-06-10_11:08:11-teuthology:no-ceph-main-distro-default-smithi failed: b"fatal: could not read Username for 'https://github.com': No such device or address\n"
docker-compose-teuthology_api-1  | INFO:     192.168.96.1:47482 - "POST /suite/?logs=true HTTP/1.1" 400 Bad Request

Please let me know if you see the problem on your end too.

@kamoltat
Copy link
Member

@VallariAg Yes I was able to recreate the issue, fixing ...

@kamoltat
Copy link
Member

@VallariAg ceph-repo should be https://github.com/ceph/ceph-ci.git

@VallariAg
Copy link
Member

@kamoltat ohhhh my bad! I'll test this again today.

@kamoltat kamoltat self-assigned this Jun 12, 2024
Comment on lines +479 to +497
{clickDryRun.isLoading | clickRun.isLoading | clickForcePriority.isLoading ? <CircularProgress /> : <Editor
value={logText}
readOnly={true}
highlight={(logText) => highlight(logText, languages.yaml)}
style={{
fontFamily: [
"ui-monospace",
"SFMono-Regular",
'"SF Mono"',
"Menlo",
"Consolas",
"Liberation Mono",
'"Lucida Console"',
"Courier",
"monospace",
].join(","),
textAlign: "initial",
}}
/>}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably reuse the new CodeBlock component now because that PR got merged.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep Code block seems like a good idea

src/pages/Schedule/index.jsx Show resolved Hide resolved
"--newest",
"--num",
"--limit",
"--priority",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I schedule runs, they are all end with "dead" status because for docker setup the "<config_yaml>": ["/teuthology/containerized_node.yaml"] config is needed. I hard-coded that in request and it resulted with "pass" status!
Right now, t-api input of <config_yaml> is a path (like the above example) but there's a PR open which will take in actual config text: ceph/teuthology-api#50. Nevertheless, I guess the react coded would probably be the same for both.

So we probably need to add support for config_yaml to get green runs in docker. It's a list type though which sounds complex so maybe we can implement this in next PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I never got to that part where I was looking for a passed run, so thank you for reaching that.

@VallariAg
Copy link
Member

Really exhaustive error handling, this is awesome!
I was able to schedule runs from the /schedule page and tested features with different teuthology-suite config, works great for me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Development

Successfully merging this pull request may close these issues.

Teuthology-suite
4 participants